Rendered

It would be useful to explain somewhere in the RFC that IFD refers to "Import From Derivation" to help people not familiar with the acronym understand it better.

I think this is particularly important for any user visible configuration, so I think it would be a mistake to call it enableIFD when we can use enableImportFromDerivation.

I think that for this to go through we have to fix the interaction between --store and IFD.

It would be useful to explain somewhere in the RFC that IFD refers to "Import From Derivation" to help people not familiar with the acronym understand it better.

I think this is particularly important for any user visible configuration, so I think it would be a mistake to call it enableIFD when we can use enableImportFromDerivation.

Done

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/status-of-lang2nix-approaches/14477/28

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/status-of-lang2nix-approaches/14477/29

How will this impact nix-env and other evaluations which expect evaluation to be "cheap" to provide user tools like search and package lists? What if we collaborated with language ecosystems to have them implement lockfiles we can parse and interpret, like the poetry2nix which does not require IFD?

What if we collaborated with language ecosystems to have them implement lockfiles we can parse and interpret, like the poetry2nix which does not require IFD?

Do you suggest ~ builtins.fromGoLock, etc.?

When contrasting with the motivation & and the last sentence of this RFC, that might at least be something to put into the future or alternative sections.

How will this impact nix-env and other evaluations which expect evaluation to be "cheap" to provide user tools like search and package lists? What if we collaborated with language ecosystems to have them implement lockfiles we can parse and interpret, like the poetry2nix which does not require IFD?

I don't think it would be a problem.
What we would likely do is set enableImportFromDerivation = false; by default, and then when updating the channel (or flake input in the future), we would make sure to build allImportedDerivations. nix-env and such would then do no IFD.

Also, importantly, adding lock files will not remove the need for IFD in many cases.
For example, we want an attr set of all hackage packages in Nixpkgs, and a lock file wouldn't let us avoid that in any way.
We still need the list of packages to be available at evaluation time, not build time. Otherwise you wouldn't be able to do nix shell nixpkgs#haskellPackages.pandoc.

There are however some important questions that aren't answered by @Ericson2314:

• How do we make sure that the built derivations aren't GC-ed? How would we root them?
• Disregarding the problem of GC, how would we implement the above for flake inputs? For channels, you could use an ad-hoc approach for only Nixpkgs.

I'm personally thinking that for the future, perhaps explicitly building a new Nixpkgs is the best approach. E.g. you'd make a copy of Nixpkgs where you'd replace every instance of import drv and similar with import (builtins.storePath /nix/store/...). This isn't trivial, however. I think we should explore that idea more before settling on an approach.

Nominating myself for shepherdship.

I don't think it would be a problem. What we would likely do is set enableImportFromDerivation = false; by default, and then when updating the channel (or flake input in the future), we would make sure to build allImportedDerivations. nix-env and such would then do no IFD.

Yes my thoughts exactly. The IFD stuff would start as just a few extra things, and the user that ops in to using enableImportFromDerivation = true; would be be responsible for either not minding the IFD or building allImportedDerivations first.

Do we have any other nominations?

I'm not sure if the policy first approach will solve much. Even if we have a policy, somebody has to invest time and effort to implement it and that is going to be the biggest bottleneck or even more like a blocker. Instead of starting out by making a policy, we could:

1. study and understand the problems better

2. decide on a solution

3. implement a solution demo

4. deliver the solution to nixpkgs in a small scale and prove that it actually works

5. then extend it and show that it actually works well for more than a single ecosystem

6. learn about some more unexpected shortcomings of the solution

7. fix some more problems

8. make the solution more generic and simpler to adopt by other maintainers

9. if they still don't want to adopt it -> then make a policy that forces them to do so.

TL;DR: starting with the policy first goes the opposite way I envision that process.

I agree. This seems like a much saner approach.

Another major thing is that I believe, having reproducible code generation for all ecosystems is quite hard to achieve and not really important.

If it is in fact a goal of the RFC to make nixpkgs suitable for the needs of people coming from language specific package managers (as stated in the motivation section), then reproducible code generation is one of the least important aspects of it.

What's important for most users is that there is a reliable way of translating their project to nix at all. I think most users wouldn't care less if their code or lock file is generated reproducibly or not. Because once it's generated, it will be reproducible anyways.

Yes! Thank you for voicing that, because I very much did not succeed in bringing that up. I'll decompose it into a problem statement (as I see it):

1. Nix upholds strong guarantees of reproducibility: if something is pinned and it works on one machine, it will on the other
2. Nix unifies pipeline: you use the same source of truth for development tools, pre-commit checks, lints, libraries, CI. You add dependencies with Nix, you build with Nix, you check with Nix and you build Docker containers with Nix (more optimized than you could by hand!)
3. Those factors make Nix very desirable for developers and devops engineers, because it removes a lot of the commonplace complexity
4. Adopting Nix is hard: you have to jump through numerous hoops to get all of that, and it gets harder the more your language benefits from Nix; packaging Rust or Go is fairly easy, but they also have a unified pipeline and strong reproducibility; Python has none, and it's hell to package.

This boils down to having a good module system for packages, version independent build logic (aka. community overrides), good customization interfaces, good user interfaces etc., none of which seems to be a goal of this RFC.

Yeah, good point. I suppose it won't help developers too much to have an easy way to migrate their project to Nix, because if something breaks, developers will have to learn about overlays, overrides, deep-replacing and other fun Nix internals. I would probably quit Nix if I were in that position, because the promise of "easy to use" had been broken once the first issue arose (even if a solution is trivial, like downgrading a dependency!)

I'll write it in a separate comment for visibility, but I think we need to decide if we want this RFC for Nixpkgs maintainers or for regular developers that are interested in Nix.

1. If we want this RFC for Nixpkgs maintainers, how much would it actually help? My guess is "not much", because the current solutions work despite being ugly, and developing a solution with feature parity is a hard problem.
2. If we want this RFC for developers, should we close this RFC? We already have a solution for developers (*2nix tooling), and it's in nix-community. I think it's good enough at this point, given that we can't upstream it to Nixpkgs.

So basically, what I'm trying to say is, would us standardizing the autogen code be valuable and feasible at this point? I now have my doubts about both; perhaps a better way to go about it would be from bottom-up. First developing the tooling that autogens code in standardized way (dream2nix), and then having an RFC to adopt it in the Nixpkgs. What do you say?

developing a solution with feature parity is a hard problem.

And, separately, unifying quite different approaches of different ecosystems would probably lead to a leaky abstraction.

@amjoseph-nixpkgs

We made an exception for Haskell; they're allowed to check in generated code. I guess we could start an RFC to make a second narrowly-defined exception.

Apologies if I get it wrong since I don't know much about this, but is the exception related to the fact the Haskell community already has the concept of a closure of dependencies at specific versions that are all compatible with each other, and a non-nix implementation of it, https://www.stackage.org, that translating that into a package set in nixpkgs made sense, and made it relatively nix idiomatic, when compared to other language ecosystems?

That would be very different from consuming third party packages from other language ecosystems, where there isn't a single dependency closure across the ecosystem, but rather each package resolves its own dependency closure.

is the exception related to ... https://www.stackage.org

No.

There are only 2,613 packages in stackage but 17,556 packages in the auto-generated ./pkgs/development/haskell-modules/hackage-packages.nix. If stackage were the source of the exception it would cover less than 15% of the auto-generated material. Obviously we're allowing the other 85% for some non-stackage reason.

More details on the role of Stackage in pkgs.haskell curation. Note in particular the bit about how we curate newer versions when Stackage is lagging behind Hackage.

Right, but I mean:

1. a single dependency closure of package versions across an entire language ecosystem, with language packages mapped 1:1 to nixpkgs packages
2. separate dependency closures for each package exposed as a nix package, with language package dependency closures mapped 1:1 to nixpkgs packages

are two completely different things, so the existence of pkgs.haskell, and its use of code generation, is an example of 1, and could be used to justify further instances of that pattern, but not necessarily to justify code generation to be used for 2.

By thinking of it that way, maybe we can make the new RFC more focused on the specific problem we want to solve.

Again sorry if I'm saying something that doesn't make sense, I'm missing a lot of context, but I do have interest in 1 specifically, so I'm trying to figure out if this (or the next) RFC is the right place to have the discussion on that idea.

Again sorry if I'm saying something that doesn't make sense, I'm missing a lot of context, but I do have interest in 1 specifically, so I'm trying to figure out if this (or the next) RFC is the right place to have the discussion on that idea.

You are making a lot of sense. I'm specifically talking about 2. I think the difference between 1 and 2 is a major one, and it requires completely different ways to think and solutions, so I would be in favor of creating a new RFC for 1 specifically, as I think the current RFC text is very 2-favored but errs on the safe side of 1 (there could also be an argument to splitting the current RFC into separate RFC 1 and RFC 2).

A complete rewrite is basically like replacing the RFC with another RFC.
If we agree that this needs to be done, then let's please close the RFC as rejected, and create a new one, as this is the least painful way of doing a re-write.

It seems recent discussion is regarding the process and there is a proposal to close and re-write/re-start. The shepherd team should meet and decide if this is the course of action. If a meeting is too much, let's at least establish a clear proposal and decision asynchronously.

Pinging everyone involved!

As discussed a little bit earlier, I propose to close this RFC as rejected. After that, two separate RFCs will be put out (sorry if the names are bad, just an example!):

1. Refactoring codegen in Nixpkgs
2. Bridging language ecosystems with Nixpkgs

The first RFC will take care of bettering the current codegen infrastructure in Nixpkgs: pkgs.haskell, Vim plugins, etc. This is a very hands-on task, and one I don't feel like I'm knowledgeable about to lead

The second RFC will basically be a draft for quite some time. @DavHau summarized very well the conditions that must be met in order for the bridging to be effective. And actually, the RFC can be much less ambitious, too! If we can get a good UI/UX going, then we don't even have to change Nixpkgs itself - we can focus on guiding users to use *2nix tooling for their own projects and not talk about Nixpkgs at all, or we could put out separate registries for different package ecosystems, or... a lot of things

I feel like the spirit of the initial RFC text was about the RFC number 2, and iterated rewrites lead to it becoming a confused mix of both. I think it would be for the best to split into two new RFCs, because while the discussion points in this PR are invaluable, it is becoming harder and harder to navigate the conversation due to the confusion surrounding the RFC's goals

All shepherds except @L-as confirmed that they're okay with closing it, does that also sound good to you @L-as?

All shepherds except @L-as confirmed that they're okay with closing it, does that also sound good to you @L-as?

Yes, sorry for not noticing this message.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cargo-lock-considered-harmful/49047/3

Sorry for necroposting.

This was discontinued and no work was done after rejecting this RFC, right?

I created NixOS/nixpkgs#336137 and would like to improve ecosystem of updaters in nixpkgs. I could also start implementing technical part of this RFC (I mean, I didn't read the RFC as it's rejected, but would love to help with future RFC)

If there is no work done on a new RFC, I could try to make it. Though writing in English is not my strong side at all

CC @Ericson2314 @L-as @sternenseemann @tomberek @DavHau